Image processor, storage medium storing an image processing program and vehicle-mounted terminal

ABSTRACT

An image processor ( 3 ) includes: an image input unit ( 7 ) that obtains an image captured by an imaging device ( 2 ) installed on a vehicle; a vehicle information input unit ( 8 ) that obtains the distance between the vehicle and a road junction or curve from a navigation system ( 5 ); a recording unit ( 15 ) in which imaging information and driver information are pre-stored; a magnifying object recognition unit ( 9 ) that recognizes a certain magnifying object in the obtained captured image; and a composing unit ( 11 ) that produces, when a magnifying object is recognized, a composite image of an magnified image and the captured image. The composing unit ( 11 ) uses the distance obtained by the vehicle information input unit ( 8 ) as well as the imaging information and the driver information stored in the recording unit ( 15 ) to calculate an area in the captured image that is not an image of a blind spot for the driver, and produces the composite image such that the magnified image is superimposed on the non-blind spot area. This allows the driver to perceive information on the blind spot area and on the magnified image through a single action.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior International Patent Application No. PCT/JP2008/053273, filed onFeb. 26, 2008, the entire contents of which are incorporated herein byreference.

FIELD

The present invention relates to an image processor that processes animage captured by a vehicle-mounted camera, etc. to make the imagedisplayable to a user.

BACKGROUND

In recent years, systems using vehicle-mounted cameras (e.g., BlindCorner Monitor: BCM) have been introduced for the purpose of preventingcollision accidents, etc., upon entry into out-of-sight intersections, Tjunctions, etc.

In BCM, for example, a blind spot area not directly observable to adriver from the driver seat is captured by a camera installed in thefront end or the like of the vehicle and the area is displayed on avehicle-mounted monitor. With BCM, it is possible to visually assist thedriver.

However, when a vehicle enters a road at an angle other than 90°, suchas entering an intersection where the intersecting roads are notperpendicular to each other, road conditions on the left and right sidesof the vehicle may not be simultaneously displayed on the BCMvehicle-mounted monitor. That is, there may arise blind spots that donot appear even on the BCM monitor. Further, in some cases, utilitypoles, pedestrians, etc., may become shielding objects, whereby thedriver may not be able to grasp the left and right conditions even bylooking at the image on the BCM monitor. In such a case, the driver maynot be aware of an approaching vehicle, meaning that road safety may notbe confirmed to a sufficient degree with BCM alone.

Meanwhile, as a means for preventing accidents, mirrors (corner mirrors)that reflect information on areas that are blind spots for drivers areplaced at intersections.

An obstacle detection system using such mirrors installed on roadsideshas been proposed (see Japanese Laid-open Patent Publication No.2007-69777 (Patent document 1), for example). In this system, infraredlight is irradiated from a vehicle to a reflector installed on aroadside and the presence or absence of a dangerous object is determinedbased on an image produced from the infrared light reflected on thereflector. When a dangerous object is present, the system notifies thedriver as such.

Further, there has also been proposed a system in which a corner mirroris identified from an image captured by a camera installed in the frontend of a vehicle and a magnified image of the corner mirror is displayedon a HUD (Head-Up Display) (see Japanese Laid-open Patent PublicationNo. 2007-102691 (Patent document 2), for example).

Patent document 1: Japanese Laid-open Patent Publication No. 2007-69777Patent document 2: Japanese Laid-open Patent Publication No. 2007-102691

However, depending on conditions such as the size, shape and directionof each corner mirror and road width, areas (blind spots) that may notbe captured even by corner mirrors may arise. For this reason, driversmay not be able to confirm road safety to a sufficient degree withcorner mirrors and their magnified image alone.

As described above, since the situation in blind spot areas changesmomentarily, it is necessary for drivers to check at all times both theBCM image and corner mirrors installed on roads or an image thereof. Forexample, immediately before entering an intersection, the driver maymake movements such as taking a look at mirrors placed on the road tocheck information on blind spot areas captured by the mirrors and takinga look at the monitor to check information on blind spot areas capturedby a BCM camera. Such actions are a burden on drivers driving vehicles.

That is, even if the environment for presenting blind spot areas and amagnified image to drivers through BCM and corner mirrors is put intoplace, when the drivers cannot check information on the blind spots andon the magnified image through a single action (action such as checkingthe monitor), the benefits of the information are halved.

For this reason, a mechanism that allows drivers to perceive theinformation on both the blind spot areas and the magnified image througha single action and to make full use of the both information is desired.

SUMMARY

According to an aspect of the invention, the image processor includes:an image input unit that obtains an image captured by an imaging deviceinstalled on a vehicle; a vehicle information input unit that obtains adistance between the vehicle and a road junction or curve in a travelingdirection of the vehicle based on information received from avehicle-mounted device installed in the vehicle; a recording unit inwhich imaging information indicating properties of the imaging deviceand driver information regarding a visual field of a driver of thevehicle are stored; a magnifying object recognition unit that recognizesa certain magnifying object in the captured image; and a composing unitthat produces, when the magnifying object recognition unit recognizesthe certain magnifying object, a composite image of a magnified image ofthe magnifying object and the captured image. The composing unit usesthe distance obtained by the vehicle information input unit as well asthe imaging information and the driver information stored in therecording unit to determine an area in the captured image that is not animage of a blind spot for the driver, and produces the composite imagesuch that the magnified image is superimposed on the non-blind spotarea.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWING(S)

FIG. 1 is a functional block diagram illustrating a configuration of anentire vehicle-mounted system including an image processor.

FIG. 2A is a simplified perspective view illustrating a vehicle 10equipped with the vehicle-mounted system.

FIG. 2B is a diagram illustrating a horizontal area to be captured by avehicle-mounted camera.

FIG. 3 is a diagram illustrating an exemplary image captured by thevehicle-mounted camera.

FIG. 4 is a diagram illustrating an example when the vehicle enters aroad at an angel other than 90°.

FIG. 5 is a diagram illustrating an example when the vehicle enters anintersection at which a shielding object is present on the left side.

FIG. 6 is a conceptual diagram illustrating an area checkable to adriver with the use of road mirrors.

FIG. 7A is a diagram illustrating an exemplary image inputted from thecamera to the image processor.

FIG. 7B is a diagram illustrating an exemplary composite image producedby the image processor.

FIG. 8 is a flowchart illustrating an exemplary operation of the imageprocessor.

FIG. 9 is a top view illustrating an example of distances L, l₀,intersection position K1, etc.

FIG. 10 is a flowchart illustrating exemplary processing performed bythe composition control unit 14.

FIG. 11 is a top view illustrating exemplary imaging area and visualfield area present when the vehicle is about to enter an intersection.

FIG. 12 is a diagram illustrating an image captured by the camera in theexample of FIG. 11 made to fit an image display area of a monitor.

FIG. 13 is a diagram illustrating an example when a composite imageincluding a magnified image with a magnified size is displayed.

FIG. 14 is a top view illustrating exemplary imaging area and visualfield area when the vehicle is about to enter an intersection.

FIG. 15A is a top view illustrating a situation in which the vehicleenters a road at an angle other than 90°.

FIG. 15B is a magnified view of the vehicle-mounted camera 2 portion.

FIG. 16 is a top view illustrating a situation in which the vehicleheads for a curve.

DESCRIPTION OF EMBODIMENT(S)

In the configuration described above, with the use of the distancebetween the vehicle and the road junction or curve in the travelingdirection of the vehicle as well as the properties of the imaging deviceand the information regarding the visual field of the driver, thecomposing unit may determine an area in the image captured by theimaging device that is not an image of a blind spot for the driver(i.e., non-blind spot area). Consequently, the composing unit mayproduce a composite image in which a magnified image of a magnifyingobject is superimposed on the non-blind spot area in the captured image.When the composite image is displayed, the driver may check the image ofblind spots in the captured image and the magnified image of themagnifying object simultaneously simply by glancing at the compositeimage. In other words, the driver may check both the blind spotscaptured by the imaging device and the magnifying object through asingle action (act of looking at the displayed composite image).

According to the present invention, it allows a drivers to recognizeinformation on blind spot areas and on a magnified image through asingle action.

In one embodiment of the invention, the composing unit may use thedistance between the vehicle and the junction or the curve and theimaging information to calculate an imaging area to be captured by theimaging device in the vicinity of a position of the junction or thecurve, use the distance between the vehicle and the junction or thecurve and the driver information to calculate a visual field area to bein the visual field of the driver in the vicinity of the position of thejunction or the curve, and use a positional relationship between theimaging area and the visual field to determine a position of thenon-blind spot area in the captured image.

As described above, the composing unit calculates both the imaging areaof imaging device and the visual field area of the driver at a positionbased on the junction or curve position. The positional relationshipbetween the two areas seems to correspond to the positional relationshipbetween the captured image and the non-blind spot area. Consequently,with the use of the positional relationship between the imaging area andthe visual field area, the composing unit may calculate the non-blindspot area in the captured image with precision.

In one embodiment of the invention, the composing unit may use thedistance obtained by the vehicle information input unit to determine amagnification level of the magnified image.

Consequently, the magnification level of the magnified image of roadmirrors may be adjusted in accordance with the distance between thevehicle and the junction or curve. As a result, a composite image thatallows the driver to check the road mirrors and blind spots in thecaptured image more easily is produced.

In one embodiment of the invention, the image input unit may furtherobtain a horizontal rudder angle of the imaging device at a time ofcapturing the captured image, and the composing unit may use thehorizontal rudder angle to calculate the imaging area.

Here, the horizontal rudder angle of the imaging device refers to anamount expressed by a rotating angle of the optical axis of the imagingdevice installed on the vehicle from a certain position when the opticalaxis is rotated about the axis of the vertical direction from thecertain position within the horizontal plane. Consequently, even with animage captured by an imaging device by rotating the optical axis of theimaging device in the horizontal direction, the composing unit maycalculate the non-blind spot area appropriately in accordance with theamount of rotation.

In one embodiment of the invention, the vehicle information input unitmay further obtain a value representing a curvature of the curve in thetraveling direction of the vehicle, and the composing unit uses thecurvature to determine the magnification level of the magnified image.

Consequently, the magnification level of the magnified image of roadmirrors may be adjusted in accordance with the curvature of a curve inthe traveling direction of the vehicle. As a result, a composite imageincluding a magnified image of road mirrors having an appropriate sizeaccording to the curvature of a curve is produced.

In one embodiment of the invention, the vehicle information input unitmay further obtain information indicating a frequency of occurrence ofaccidents at the junction or the curve, and the composing unit may usethe frequency of occurrence of accidents to determine the magnificationlevel of the magnified image.

Consequently, the magnification level of the magnified image of roadmirrors may be adjusted according to the frequency of occurrence ofaccidents at a junction or curve in the traveling direction of thevehicle. As a result, a composite image including a magnified image ofroad mirrors having an appropriate size according to the degree of riskat a junction or curve is produced.

In one embodiment of the invention, the image processor may work inconjunction with a navigation system installed in the vehicle, and thevehicle information input unit may obtain the distance between thevehicle and the road junction or curve in the traveling direction of thevehicle from the navigation system installed in the vehicle. Byconfiguring the image processor to be able to work in conjunction withthe navigation system in this way, it is possible to perform processingin an efficient manner.

In one embodiment of the invention, the vehicle information input unitmay further obtains information indicating presence or absence of themagnifying object at the junction or the curve from the navigationsystem, and the magnifying object recognition unit may recognize acertain magnifying object in the obtained captured image when themagnifying object is present at the junction or the curve.

As described above, by determining whether to recognize the magnifyingobject or not based on information indicating the presence or absence ofthe magnifying object at the junction or curve, the recognition may beperformed precisely and efficiently.

In one embodiment of the invention, when the distance becomes equal toor less than a certain distance, the magnifying object recognition unitmay recognize a certain magnifying object in the obtained capturedimage.

According to an aspect of the invention, an image processing methodexecuted by a computer includes: obtaining an image captured by animaging device installed on a vehicle; obtaining a distance between thevehicle and a road junction or curve in a traveling direction of thevehicle based on information from a vehicle-mounted device installed inthe vehicle; reading and obtaining imaging information indicatingproperties of the imaging device and driver information regarding avisual field of a driver of the vehicle stored in a recording unitaccessible to the computer; recognizing a certain magnifying object inthe captured image; and producing, when the certain magnifying object isrecognized in the magnifying object recognition, a composite image of amagnified image of the magnifying object and the captured image. In theproduction of the composite image, the obtained distance as well as theobtained imaging information and driver information are used todetermine a non-blind spot area in the captured image that does notinclude an image of a blind spot for the driver, and the composite imageis produced such that the magnified image is superimposed on thenon-blind spot area.

According to an aspect of the invention, a storage medium storing animage processing program that causes a computer to perform processingof: obtaining an image captured by an imaging device installed on avehicle; obtaining a distance between the vehicle and a road junction orcurve in a traveling direction of the vehicle based on information froma vehicle-mounted device installed in the vehicle; obtaining imaginginformation indicating properties of the imaging device and driverinformation regarding a visual field of a driver of the vehicle storedin a recording unit accessible to the computer; recognizing a certainmagnifying object in the captured image; and producing, when the certainmagnifying object is recognized in the magnifying object recognition, acomposite image of a magnified image of the magnifying object and thecaptured image. In the production of the composite image, the obtaineddistance as well as the obtained imaging information and driverinformation are used to determine a blind-spot area in the capturedimage that does not include an image of a blind spot for the driver, andthe composite image is produced such that the magnified image issuperimposed on the non-blind spot area.

According to an aspect of the invention, a vehicle-mounted terminal iscapable of working in conjunction with an imaging device installed on avehicle, and the vehicle-mounted terminal includes: a navigation systemhaving a function of identifying a position of the vehicle and includinga map data recording unit in which road information including a positionof each road junction or curve is stored; an image input unit thatobtains an image captured by the imaging device; a vehicle informationinput unit that obtains a distance between the vehicle and a roadjunction or curve in a traveling direction of the vehicle with the useof the position of the vehicle identified by the navigation system andthe road information; a recording unit in which imaging informationindicating properties of the imaging device and driver informationregarding a visual field of a driver of the vehicle are stored; amagnifying object recognition unit that recognizes a certain magnifyingobject in the captured image; a composing unit that produces, when themagnifying object recognition unit recognizes the certain magnifyingobject, a composite image of a magnified image of the magnifying objectand the captured image; and a display unit that displays the compositeimage produced by the composing unit.

Hereinafter, one embodiment of the present invention will be describedwith reference to the drawings.

<Overview of Configuration of Vehicle-mounted System>

FIG. 1 is a functional block diagram illustrating a configuration of anoverall vehicle-mounted system, including an image processor accordingto this embodiment. A vehicle-mounted system 1 illustrated in FIG. 1 isa system installed in a vehicle and the system includes a camera(imaging device) 2, an image processor 3, a GPS antenna 4, a navigationsystem (navigation system unit) 5 and a monitor (display unit) 6. Thecamera 2 is placed at a position from which a front view from thevehicle may be captured. The image processor 3 receives and processes animage of the front view captured by the camera 2 and outputs theprocessed image to the monitor 6.

The GPS antenna 4 receives radio waves from a plurality of GPSartificial satellites (GPS satellites). The navigation system 5 measuresthe current position of the vehicle based on the radio waves received bythe GPS antenna 4. The navigation system 5 produces navigationinformation using the current position and map data pre-stored in a mapdata recording unit 17 and displays the produced information on themonitor 6. In addition to a road map (including information on roadwidths and positions of junctions and curves), the map data alsoincludes data on a variety of facilities, landmarks and the like. Themap data is used by the navigation system 5 in displaying the currentposition, route search and route guidance, for example.

The image processor 3 receives information on intersections or curves inthe traveling direction, the current position of the vehicle and thelike from the navigation system 5 and use them in image processing. Theimage processor 3 includes an image input unit 7, a vehicle informationinput unit 8, a magnifying object recognition unit 9, a composing unit11, an output unit 15 and a recording unit 16. The composing unit 11includes a magnifying unit 12, a superimposition unit 13 and acomposition control unit 14. Details on the inner workings of the imageprocessor 3 will be described later.

<Mounting Example on Vehicle>

FIGS. 2A and 2B illustrate an exemplary configuration of thevehicle-mounted system 1 when the system is mounted on a vehicle. FIG.2A is a perspective view illustrating a schematic configuration of avehicle 10 on which the vehicle-mounted system 1 is mounted. The camera2 is installed in the front end of the vehicle 10 and is connected to ahousing (vehicle-mounted terminal) 18. The navigation system 5, theimage processor 3 and the monitor 6 are integrated to form the housing18. The monitor 6 is formed at a position that may be observed by adriver H1 in the vehicle 10.

For example, the housing 18 includes a computer including a CPU,recording media (RAM, ROM, HDD, etc.), a display, a power circuit, buslines for connecting these components, and the like. The navigationsystem 5 as well as the image input unit 7, the vehicle informationinput unit 8, the magnifying object recognition unit 9, the composingunit 11 and the output unit 15 of the image processor 3 are eachfunctionally implemented via execution of a certain program by the CPU.The program for implementing each of the functions and a recordingmedium in which the program is stored are also one example of theembodiment of the present invention. Herein, the recording medium isnon-transitory tangible medium and dose not include transitory mediumsuch as propagating signal per se. Further, the recording unit 16 andthe map data recording unit 17 are implemented via the recording mediumincluded in the computer.

Note the example illustrated in FIG. 2A does not limit forms in whichthe vehicle-mounted system 1 is mounted on the vehicle. For example, thenavigation system 5 and the monitor 6 may form a single housing, and theimage processor 3 may be installed as an ECU (Electronic Control Unit)to be connected to the camera 2 and the housing. Further, the imageprocessor 3 may be formed with such a chip as a 1394 controller LSI.Further, the monitor 6 may be formed with an instrumental panel, an HUDor the like. In FIG. 2A, a range between two lines m1 and n1 indicatesan area to be captured by the camera 2 in the vertical direction.

FIG. 2B is a diagram illustrating an area to be captured by the camera 2of the vehicle 10 in the vertical direction when the vehicle 10 is aboutto enter an intersection. In FIG. 2B, an area between two lines p2 andn2 indicates a horizontal area to be captured by the camera 2. In theexample illustrated in FIG. 2B, an angle α between the two lines p2 andn2 is a little less than 180° (hereinafter, an area expressed in angle,which is captured by the camera and displayed on the monitor 6, such asthe angle α, will be referred to as a monitorable angle). Here, themonitorable angle α of the camera 2 is close to 180°. Thus, when thefront end of the vehicle 10 enters the intersection, an image of theleft and right sides of the road perpendicular to the travelingdirection of the vehicle 10 is to be captured by the camera 2 anddisplayed on the monitor 6. Further, since the camera is positioned inthe front end of the vehicle 10, the driver H1 may check blind spotsahead of the vehicle 10 right away when the vehicle 10 moves forward.

FIG. 3 is a diagram illustrating an exemplary image captured by thecamera 2 of the vehicle 10 that is about to enter an intersection. Inthe image illustrated in FIG. 3, areas that are blind spots for thedriver H1 are in a right end portion AR and a left end portion AL.

In the example illustrated in FIG. 2B, the left and right sides of thevehicle 10 are captured by a single camera whose horizontal monitorableangle is close to 180°. Alternatively, two cameras each having amonitorable angle of about 90° may be installed on the vehicle 10 toface the left and right directions, respectively. Further, by placingthree cameras in the front end of the vehicle in the front, left andright directions, the monitorable range may be increased to more than180°. The number of cameras to be installed is not particularly limitedand is determined in view of the cost, image viewability and the like asneeded.

Further, the camera installation position is not limited to the positionillustrated in FIGS. 2A and 2B. For example, the camera may also beinstalled on the hood, rearview mirror, door mirror, etc., of thevehicle 10. Further, in order to capture a rearview image when thevehicle 10 moves backward, the camera may be installed on the rear ofthe vehicle 10. In this embodiment, a case of capturing a front viewimage by a single camera will be described as an example.

<Exemplary Output Image from Image Processor>

When the vehicle 10 enters an intersection or T junction, left and rightconditions may not be captured by the camera 2 in some cases. Forexample, as illustrated in FIG. 4, when the vehicle 10 enters a road atan angle other than 90° and there is a fence 21 on the right side of thevehicle 10, not the road conditions in the right direction but the fence21 appears in an image captured by the camera 2 at the right end. Thatis, because a vehicle 22 a approaching from the right side isovershadowed by the fence 21, the vehicle 22 a does not appear in theimage captured by the camera 2. Also, a vehicle 22 b approaching fromthe left side does not appear in the image captured by the camera 2unless the vehicle 22 b enters into the monitorable angle of the camera2.

Further, as another example, as illustrated in FIG. 5, also when thevehicle 10 enters an intersection at which a shielding object 23 (e.g.,a utility pole or pedestrian) is present on the left side, a vehicle 22c approaching from the left side does not appear in an image captured bythe camera 2 because the vehicle 22 c is overshadowed by the shieldingobject 23. In this way, there are situations where road conditions ofthe left and right sides may not be captured by the camera 2.

Meanwhile, the driver H1 may look at a road mirror (commonly known as acurve mirror) placed at an intersection, T junction or curve to checkroad conditions that are not directly observable to the driver. FIG. 6is a conceptual diagram illustrating an area where the driver H1 maycheck with the use of road mirrors when the vehicle 10 is about to entera T junction. In the example illustrated in FIG. 6, two road mirrors 24a, 24 b are placed at the T junction. A range between lines p3 and n3indicates an area where the driver H1 of the vehicle may check by takinga direct look. The driver H1 may look at the road mirror 24 a to check arange between lines p4 and n4 and look at the road mirror 24 b to checka range between lines p5 and n5.

In the example illustrated in FIG. 6, areas S1 as blind spots for thedriver H1 on the left and right sides are captured by the camera 2 andare displayed on the monitor 6. In this case, the driver H1 may look atthe road mirrors and the monitor 6 to check left and right conditionsthat are not directly observable to the driver H1. The image processor 3according to this embodiment provides an image that allows the driver H1to check both the blind spot areas S1 and the areas reflected on theroad mirrors 24 a, 24 b in a situation as illustrated in FIG. 6 bysimply looking at the monitor 6, for example.

FIG. 7A is a diagram illustrating an exemplary image inputted from thecamera 2 to the image processor 3. The image processor 3 produce acomposite image by superimposing on the original image a magnified imageof a road mirror portion A1 in the image illustrated in FIG. 7A. FIG. 7Bis a diagram illustrating an exemplary composite image produced by theimage processor 3. As illustrated in FIG. 7B, the image processor 3 mayproduce an image in which a magnified image is superimposed on anon-blind spot area (area that is not included in the blind spot areasS1) in the original image (FIG. 7A).

In this way, by superimposing a magnified image of road mirrors on thenon-blind spot area for the driver H1 in the original captured image anddisplaying the composite image, the overall blind spot area for thedriver H1 may be displayed in a single screen. Hereinafter, exemplaryconfiguration and operation of the image processor 3 capable ofperforming such image processing will be described in detail.

<Configuration of Image Processor 3>

In the image processor 3 illustrated in FIG. 1, the image input unit 7obtains image data of a captured image from the camera 2 and makes thedata accessible to the magnifying object recognition unit 9 and thecomposing unit 11. The image input unit 7 subjects the image signalsreceived from the camera 2 to A/D conversion and other necessaryconversion processing and records the signals frame by frame in arecording medium accessible to the magnifying object recognition unit 9and the composing unit 11. The image input unit 7 may also receive imagedata that has already been subjected to necessary conversion processing,such as A/D conversion.

The magnifying object recognition unit 9 reads the image data obtainedby the image input unit 7 frame by frame and determines the presence orabsence of an area that may be recognized as a magnifying object (a roadmirror in this case) in each frame of the image. When an area that maybe recognized as a road mirror is present, the magnifying objectrecognition unit 9 extracts the data of the area and passes the data tothe magnifying unit 12. The magnifying unit 12 magnifies an image of thearea to produce a magnified image.

With the use of known image recognition techniques, the magnifyingobject recognition unit 9 may recognize a road mirror portion in theimage. As an example, first, the magnifying object recognition unit 9uses a Laplacian filter to extract an edge portion in the image. Then,the magnifying object recognition unit 9 matches image data forming theedge portion with pre-stored feature quantity data of a road mirror(e.g., a template of a standard road mirror) and calculates thecorrelation value. An area in which the correlation value is larger thana threshold value may be determined as a road mirror area.

Further, templates of objects that could be easily misidentified as aroad mirror, such as road signs, may be pre-stored as feature quantitydata. In this case, the magnifying object recognition unit 9 may beconfigured not to recognize an object as a road mirror when thecorrelation value between such templates and image data forming the edgeportion is larger than a threshold value. As a result, the precision ofthe recognition improves. Note that the road mirror recognition is notlimited to the example described above.

As described above, magnifying objects recognized by the magnifyingobject recognition unit 9 are objects that need to be magnified andpresented to the driver. That is, objects that provide the driver withinformation beneficial to the driver in driving vehicle are predefinedas magnifying objects. In the example described above, the pre-storedfeature quantity data defines road mirrors as magnifying objects.Magnifying objects are not limited to road mirrors and road signs,guideboards and road surface markings (letters (e.g., “STOP”) and arrowspainted on a road surface) may also be defined as magnifying objects.

The magnifying unit 12 magnifies the image of the road mirror areaaccording to a value representing a magnification level received fromthe composition control unit 14. The magnifying unit 12 passes themagnified image to the superimposition unit 13.

The superimposition unit 13 superimposes the received magnified image onthe frames (hereinafter referred to as the original frames)corresponding to the image data of the captured image obtained by theimage input unit 7 to produce a composite image. When performingsuperimposition, the superimposition unit 13 obtains from thecomposition control unit 14 information indicating the superimpositionposition of the magnified image in the original frames, and superimposesthe magnified image on the original frames based on the obtainedinformation.

In this way, the composite image produced by the superimposition unit 13is outputted to the monitor 6 via the output unit 15. Frames in whichthe magnifying object recognition unit 9 has not found a mirror portionare sent to the output unit 15 without being processed by thesuperimposition unit 13 and are displayed on the monitor 6.

In the recording unit 16, imaging information indicating the propertiesof the camera 2 installed on the vehicle 10 and driver informationregarding the visual field of the driver H1 of the vehicle 10 arepre-stored. The imaging information includes information used indetermining an area to be captured by the camera 2. For example, theimaging information includes such information as the monitorable angle,angle of view, lens properties of the camera 2 and the installationposition of the camera 2 on the vehicle 10.

Further, the driver information includes information that enables toestimate the visual field of the driver sitting in the driver seat ofthe vehicle 10. For example, the driver information includes suchinformation as the position of the driver's eyes in the vehicle 10 andthe visual field properties (e.g., effective visual field) of thedriver. The driver information is not limited to pre-stored fixedvalues. For example, the vehicle information input unit 8 may receiveinformation on the visual field of the driver from a vehicle-mounteddevice (not shown) for monitoring the driver's eye movements and storethe information in the recording unit 16 as a piece of the driverinformation.

The vehicle information input unit 8 obtains the current position of thevehicle 10 and the immediate junction position in the travelingdirection of the vehicle 10 from the navigation system 5, calculates thedistance between the vehicle 10 and the junction (hereinafter referredto as distance L) and notifies the composition control unit 14 of thecalculated distance.

The way to obtain the distance L between the vehicle 10 and the junctionis not limited to one described above. The vehicle information inputunit 8 may obtain the distance L based on data received from avehicle-mounted device capable of gathering information for determiningthe current position. The vehicle-mounted device is not limited to aparticular device. For example, the vehicle information input unit 8 mayreceive data indicating the distance L between the vehicle 10 and thejunction from the navigation system 5 or may calculate the distance Lusing radio waves received by the GPS antenna 4 and the map data storedin the map data recording unit 17. Further, in addition to or in placeof the distance between the vehicle 10 and the junction, the vehicleinformation input unit 8 may obtain the distance between the vehicle andthe curve in the traveling direction. Further, the vehicle informationinput unit 8 may further use information received from a vehicle speedsensor, a vehicle direction sensor, etc., (all of which are not shown)to determine the current position of the vehicle 10.

The composition control unit 14 uses the distance L between the vehicle10 and the junction as well as the imaging information and the driverinformation stored in the recording unit 16 to calculate themagnification level of the road mirror area in the image and theposition at which the magnified image is superimposed, and notifies themagnifying unit 12 and the superimposition unit 13 of the results,respectively. Here, the composition control unit 14 calculates an imagearea in the original frames of the captured image, area directlyobservable to the driver H1 (=a non-blind spot area that does notinclude an image of a blind spot for the driver H1), and calculates thesuperimposition position so that the magnified image is superimposed onthe non-blind spot area.

For example, the composition control unit 14 uses the distance L and theimaging information to calculate an area to be captured by the camera 2(imaging area) in the vicinity of a position apart from the vehicle 10by the distance L. Further, the composition control unit 14 uses thedistance L and the driver information to calculate an area to be in thevisual field of the driver (visual field area) in the vicinity of theposition apart from the vehicle 10 by the distance L. And thecomposition control unit 14 uses the positional relationship between theimaging area and the visual field area to calculate the non-blind spotarea in the original frames of the captured image. The compositioncontrol unit 14 may calculate the non-blind spot area such that thepositional relationship of the imaging area with the visual field areacorresponds to the positional relationship of the non-blind spot areawith the original frames of the captured image. Note that the way tocalculate the non-blind spot area is not limited to one described above.The composition control unit 14 may determine the non-blind spot areafor the driver H1 using the properties of the camera 2, road shape,intersection position, vehicle position, etc.

Further, the composition control unit 14 may use information inputted bythe vehicle information input unit 8 to control the operation of theimage input unit 7 or the magnifying object recognition unit 9. Forexample, the composition control unit 14 may control the image inputunit 7 and the magnifying object recognition unit 9 to operate only whenthe distance L is smaller than a certain distance. As a result, themagnifying object recognition unit 9 performs the road mirrorrecognition every time the vehicle 10 approaches an intersection.

Further, it is not necessary to perform the road mirror recognitionevery time the vehicle 10 approaches an intersection. For example, bypre-including in the map data of the navigation system 5 informationindicating the presence or absence of a magnifying object (a road mirrorin this case) at each junction, the vehicle information input unit 8 maybe configured to obtained the pre-included information. In this case,the composition control unit 14 may control the magnifying objectrecognition unit 9 to perform the road mirror recognition when thevehicle 10 approaches a junction with a road mirror. Also, themagnifying object recognition unit 9 may directly receive thepre-included information from the vehicle information input unit 8 todetermine whether to perform the road mirror recognition or not.

<Exemplary Operation of Image Processor 3>

Next, an exemplary operation of the image processor 3 will be described.FIG. 8 is a flowchart illustrating an exemplary operation of the imageprocessor 3. In the exemplary operation illustrated in FIG. 8, first,the image processor 3 initializes “Flag” (set Flag to 0) indicatingwhether to output a composite image or not (Op1).

Then, the vehicle information input unit 8 obtains the current positionof the vehicle 10 and a position K1 of the closest intersection in thetraveling direction of the vehicle 10 from the navigation system 5(Op2). For example, the current position information and theintersection position K1 are each expressed in latitude and longitude.The vehicle information input unit 8 calculates the distance L betweenthe current position of the vehicle 10 and the intersection position K1(Op3). The composition control unit 14 is notified of the distance L.

When the distance L is smaller than a threshold value l₀, in otherwords, when the vehicle 10 is in the vicinity of the intersection (L<l₀:Yes at Op4) and has not passed the intersection (L≧0: Yes at Op5), thecomposition control unit 14 causes the image input unit 7 to obtain animage captured by the camera 2 (Op9).

FIG. 9 is a top view illustrating an example of positional relationshipsamong the distances L, l₀, the intersection position K1, etc. In theexample illustrated in FIG. 9, the intersection position K1 is set atthe center of the intersection (the intersection point of lines (centrallines) passing through the center of the intersecting roads). Thevehicle 10 is traveling towards the intersection position K1. Here, thedistance l₀ is the sum of a distance l₁ determined by the width of theroad (road width A) into which the vehicle 10 is about to enter and acertain fixed value l₂. The distance l₁ is the distance between theintersection position and the roadside, and may be determined by l₁=A/2,for example. In this way, it is possible to calculate the thresholdvalue l₀ used in determining the time period over which an image iscaptured by the camera 2 with the addition of the value based on theroad width A to the fixed value performed by the composition controlunit 14.

At Op9 in FIG. 8, the image input unit 7 obtains a single frame of thecaptured image, for example. Although an example where a single frame ofthe image is obtained will be described in the following, the imageinput unit 7 may obtain a plurality of frames at Op9 and each of theframes may be subjected to a process at Op10, which will be describedlater.

When the vehicle 10 is not in the vicinity of the intersection (No atOp4) or has already passed the intersection (No at Op8), the compositioncontrol unit 14 interprets Flag without obtaining the image captured bythe camera 14 (Op5). When Flag is 0 (“Flag=0”) (No at Op5), the processat Op 2 is performed again. When Flag is 1 (“Flag=1”) (Yes at Op5), thecomposition control unit 14 instructs the magnifying unit 12 and thesuperimposition unit 13 to end the image superimposition (Op6) and setsFlag to 0 (“Flag=0”) (Op7). Thereafter, the process at Op2 is performedagain.

When the vehicle 10 arrives at the point apart from the intersectionposition by the certain distance l₀, due to the processes at Op4 to Op9,the image input unit 7 starts obtaining the image captured by the camera2 and stops obtaining the image captured by the camera 2 when thevehicle 10 passes the intersection position.

When the image input unit 7 obtains a frame of the image captured by thecamera 2 (original frame) at Op9, the magnifying object recognition unit9 extracts from the original frame an area that may be recognized as aroad mirror (road mirror area) (Op10). When the magnifying objectrecognition unit 9 extracts the road mirror area (Yes at Op11), thecomposition control unit 14 calculates the magnification level of theroad mirror area and the position at which a magnified image issuperimposed on the original frame (Op13). At Op13, the compositioncontrol unit 14 uses the distance L calculated at Op3 as well as theimaging information and the driver information stored in the recordingunit 16 to calculate the magnification level and the superimpositionposition. In so doing, the composition control unit 14 calculates thesuperimposition position such that the magnified image is superimposedon the area in the original frame of the image that is directlyobservable to the driver H1 (non-blind spot area). Details on theprocess at Op13 will be described later.

Base on the magnification level calculated at Op13, the magnifying unit12 produces a magnified image of the road mirror area extracted at Op10(Op14). The superimposition unit 13 produces a composite image in whichthe magnified image is superimposed on the original frame based on thesuperimposition position calculated at Op13 (Op15). Then, the outputunit 15 produces display data which is processed such that the compositeimage may be displayed on the monitor 6 (Op16) and outputs the data tothe monitor 6 (Op17). As a result, the image in which the magnifiedimage of the road mirrors is superimposed on the non-blind spot area inthe image captured by the camera 2 is displayed on the monitor 6. Whenthe composite image is displayed on the monitor 6, Flag is set to 1(“Flag=1”) (Op18).

Thereafter, Op2 is performed again. As a result of the processesillustrated in FIG. 8, the image captured by the camera 2 is displayedon the monitor 6 during the period between the arrival of the vehicle 10at the point apart from the intersection position by the distance l₀ andthe passage of the intersection position. At that time, when a roadmirror is in an image captured by the camera 2, an image of the roadmirror is magnified and displayed in the non-blind spot area in thecaptured image. By simply glancing at the image on the monitor 6, thedriver H1 may check both the conditions of the blind spot area reflectedon the road mirror and the conditions of the blind spot area captured bythe camera 2.

<Exemplary Calculation of Magnification Level and Superimpositionposition>

An exemplary calculation of the magnification level and thesuperimposition position at Op13 in FIG. 8 will be described. FIG. 10 isa flowchart illustrating an example of the process performed by thecomposition control unit 14 at Op13. In FIG. 10, the composition controlunit 14 retrieves the camera properties of the camera 2 from therecording unit 16 as the imaging information (Op21). For example, thecamera properties include the installation position, angle of view andlens properties of the camera 2.

Further, the composition control unit 14 obtains the view position andeffective visual field of the driver from the recording unit 16 as thedriver information (Op22). Normally, in a case of the vehicle 10, theview position of the driver may be determined based on the driver seatposition. Thus, the driver seat position may be pre-stored in therecording unit 16 as the data indicating the view position of thedriver.

The composition control unit 14 uses the camera properties obtained atOp21 and the distance L to calculate the imaging area captured by thecamera 2 in the vicinity of the intersection position (Op22).Furthermore, the composition control unit 14 uses the driver informationobtained at Op22 and the distance L to calculate the visual field areadirectly observable to the driver in the vicinity of the intersectionposition (Op23).

Hereinafter, an exemplary calculation at Op22 and Op23 will be describedwith reference to FIG. 11. As the imaging area and the visual field areain the vicinity of the intersection, here, a case of calculating thoseon a line passing through a horizontal plane including the intersectionposition K1 and perpendicular to the traveling direction of the vehiclewill be described. Note that the imaging area and the visual field areain the vicinity of an intersection are not limited to those in thisexample. For example, the imaging area and the visual field area on aline passing through or a plane including a position advanced from theintersection position K1 by ½ of the road width A may be calculated.

FIG. 11 is a top view illustrating exemplary imaging area and visualfield area when the vehicle 10 is about to enter an intersection. In theexample illustrated in FIG. 11, the position of the camera 2 on thevehicle 10 corresponds to the current position of the vehicle 10 and theintersection position K1 is on a line extending from the currentposition to the traveling direction of the vehicle 10.

In FIG. 11, a range between lines p6 and n6 is an area to be captured bythe camera 2 in the horizontal direction. The angle “α” between thelines p6 and n6 is the monitorable angle of the camera 2. The value ofangle of view of the camera 2 included in the camera properties may beused as the value of “α”. Further, the composition control unit 14 maycalculate the value of “α” based on the angle of view and lensproperties of the camera 2. Alternatively, the value of “α” may bepre-stored in the recording unit 16 as a fixed value.

Further, a range between lines p7 and n7 is an area that is directlyobservable to the driver H1. An angle “β” between the lines p7 and n7 isthe effective visual field of the driver H1. For example, the value of“β” may be pre-stored in the recording unit 16 as a fixed value. Theeffective human visual field is a visual field in which an object can becaptured simply by eye movements and a target object may be perceived inthe noise. Since the normal effective visual field of humans is about15° in left and right eyes, this value may be stored as the value of theangle “β”.

Further, l_(m) denotes the distance between the installation position ofthe camera 2 and the driver H1 in the traveling direction and n denotesthe distance between the installation position of the camera 2 and thedriver H1 in the direction perpendicular to the traveling direction.These distances l_(m) and n may be determined from the installationposition of the camera 2 obtained at Op21 and the view position of thedriver obtained at Op22. Alternatively, these distances l_(m) and n maybe pre-stored in the recording unit 16.

On a line q, line passing through a horizontal plane including theintersection position K1 and perpendicular to the traveling direction ofthe vehicle, the area captured by the camera 2 is from the intersectionpoint of the lines q and p6 to the intersection point of the lines q andn6. Further, the area that is directly observable to the driver H1 isfrom the intersection point of the lines q and p7 to the intersectionpoint of the lines q and n7.

At Op22, the composition control unit 14 calculates ½ of the length ofthe area captured by the camera 2 on the line q (=m1) as the imagingarea. In this calculation, the monitorable angle α and the distance Lbetween the vehicle 10 and the intersection position are used.Specifically, the value of m1 may be calculated as expressed by thefollowing equations (1) and (2).

tan(α/2)=m1/L  (1)

m1=L×tan(α/2)  (2)

Further, at Op23, the composition control unit 14 calculates ½ of thelength of the area directly observable to the driver H1 on the line q(=m2) as the visual field area. In this calculation, the distance L, theangle β and the distance l_(m) between the camera 2 and the driver H1 inthe traveling direction are used. Specifically, the value of m2 may becalculated as expressed by the following equations (3) and (4).

tan(β/2)=m2/(L+l _(m))  (3)

m2=(L+l _(m))×tan(β/2)  (4)

The composition control unit 14 uses ml and m2 to calculate thepositional relationship between the imaging area and the visual fieldarea. Specifically, in the example illustrated in FIG. 11, thecomposition control unit 14 calculates a distance X between theintersection point of the lines q and p6 and the intersection point ofthe lines q and p7. The distance X is an exemplary value representingthe positional relationship between the imaging area and the visualfield area, and it is from the left end of the imaging area to the leftend of the visual field area. The value of X may be calculated by thefollowing equation (5) when l₀<L, equation (6) when l₁<L≦l₀ and equation(7) when 0<L≦l₁. In this way, by changing the ways to calculate thevalue of X in accordance with the value of the distance L between thevehicle 10 and the intersection position K1, it is possible to calculatethe superimposition position appropriately in accordance with thedistance L.

When l₀≦L:

X=0  (5)

When l₁<L≦l₀:

$\begin{matrix}\begin{matrix}{X = {{m\; 1} - \left( {{m\; 2} - n} \right)}} \\{= {\left\{ {L \times {\tan \left( {\alpha/2} \right)}} \right\} - \left\{ {{\left( {L + 1_{m}} \right) \times {\tan \left( {\beta/2} \right)}} - n} \right\}}}\end{matrix} & (6)\end{matrix}$

When 0<L≦l₁:

X={l ₁×tan(α/2)}−{(l ₁ +l _(m))×tan(β/2)−n}  (7)

With the use of m1, m2 and X calculated in this way, the compositioncontrol unit 14 calculates a value representing the non-blind spot areain the image captured by the camera 2 (Op25 in FIG. 10). For example,the composition control unit 14 calculates the left end position of aportion in the captured image where the visual field of the driver H1 isshown (non-blind spot area).

Specifically, the composition control unit 14 uses the valuerepresenting the imaging area (m1), the value representing the visualfield area (m2) and the distance X between the left end of the imagingarea and the left end of the visual field area to calculate the left endposition of the non-blind spot area in the image captured by the camera2. An example of this calculation will be described with reference toFIG. 12.

FIG. 12 is a diagram illustrating an image captured by the camera 2 inthe example of FIG. 11, which is fit into the display area of themonitor. In the example illustrated in FIG. 12, the width of the imagingarea on the line q (2×m1) corresponds to a width W1 of a captured imageG1. Here, it is assumed that the width W1 of the captured image G1 isthe width of the image display area of the monitor 6. For example, thewidth W1 is pre-stored in the recording unit 16 as a fixed value.

As illustrated in FIG. 12, it is possible to assume that the positionalrelationship between the imaging area and the visual field area on theline q corresponds to the positional relationship between the overallcaptured image G1 and the non-blind spot area in the image. In otherwords, it is possible to assume that the relationship between the widthW1 of the captured image G1 and a length X_(PIX) between the left end ofthe captured image G1 and the left end of the non-blind spot areacorresponds to the relationship between the width of the imaging area(2×m1) and the distance X. If that is the case, the following equation(8) holds true.

(2×m1):W1=X:X _(PIX)  (8)

X_(PIX) may be calculated by the following equation (9).

X _(PIX)=(X×W1)/(2×m1)  (9)

The composition control unit 14 calculates the value of X_(PIX) as thevalue representing the non-blind spot area (Op25 in FIG. 10) and furthernotifies the superimposition unit 13 of X_(PIX) as the value directlyrepresenting the superimposition position (Op26). Note that the valuerepresenting the superimposition position is not limited to X_(PIX). Forexample, the composition control unit 14 may determine the left end andright end positions of the visual field area on the line q and set theposition in the captured image corresponding to the midpoint of the twoends as the superimposition position.

The method for calculating the non-blind spot area described above isbased upon the premise that the monitorable angle α of the camera 2 issmaller than 180°. When the angle of view of the camera 2 is 180° ormore, for example, by subtracting 1° each from 180° on the left andright to set a to 178°, the non-blind spot area may be calculated usingthe above-described calculation method.

The area captured by the camera 2 becomes larger as the value of αincreases. This results in a decrease in the proportion of the image ofa road mirror portion to the captured image, making the road mirrorportion difficult to see. For this reason, by pre-storing in therecording unit 16 the maximum monitorable angle α_(max) (=thresholdvalue) of the camera 2 at which road mirrors are recognized, it ispossible to set an image of the area within the maximum monitorableangle α_(max) as the captured image even when the angle of view of thecamera 2 is larger than the maximum monitorable angle α_(max). As aresult, even when the angle of view of the camera 2 is 180° or more, thenon-blind spot area may be calculated using the calculation methoddescribed above.

Next, at Op27, the composition control unit 14 calculates themagnification size of the magnified image (the size aftermagnification). Here, as illustrated in FIGS. 7A and 7B, it is assumedthat W1 and H1 denote the width and height of the image display area ofthe monitor 6, in other words, the width and height of the capturedimage, respectively, and W2 and H2 denote the width and height of themagnified image, respectively. The values of W1 and H1 are pre-stored inthe recording unit 16, for example. The values of W2 and H2 may becalculated respectively by the following equations (10) and (11) whenl₀<L, equations (12) and (13) when l₁<L≦l₀ and equations (14) and (15)when 0<L≦l₀. In this way, by changing the ways to calculate themagnification size (W2, H2) in accordance with the value of the distanceL between the vehicle 10 and the intersection position K1, it ispossible to calculate the magnification size appropriately in accordancewith the distance L.

When l₀<L:

W2=0  (10)

H2=0  (11)

When l₁<L≦l₀:

W2=W1−(a×L)  (12)

H2=H1−(b×L)  (13)

When 0<L≦l₁:

W2=W1−(a×l ₁)  (14)

H2=H1−(b×l ₁)  (15)

In the equations (12) to (15), coefficients a, b are constants and thevalues of the coefficients a, b are appropriately determined based onthe camera properties and the road width B, for example. Specifically,the coefficient a may be determined with the use of a table in whichvalue combinations of the horizontal angle of view of the camera 2(horizontal angle of view VX) and the road width B and values of thecoefficient a corresponding to the combinations are pre-stored. Table 1below is an example of the table in which the values of the coefficienta corresponding to the value combinations of the horizontal angle ofview VX and the road width B are stored.

TABLE 1 Road width B Horizontal angle B₁ ≦ B < B_(m−1) ≦ B < of view VXB < B₁ B₂ . . . B_(m) B_(m) ≦ B VX < VX₁ a¹⁻¹ a¹⁻² . . . a_(1−(m−1))a_(1−m) VX₁ ≦ VX < VX₂ a²⁻¹ a²⁻² . . . a_(2−(m−1)) a_(2−m) . . . . . . .. . . . . . . . . . . VX_(n−1) ≦ VX < a_((n−1)−1) a_((n−1)−2) . . .a_((n−1)−(m−1)) a_((n−1) −m) VX_(n)(180°) VX_(n)(180°) ≦ VX a_(n−1)a_(n −2) . . . a_(n−(m−1)) a_(n−m)

Similarly, the coefficient b may be determined with the use of a tablein which value combinations of the vertical angle of view of the camera2 (vertical angle of view VZ) and the road width B and the values of thecoefficient b corresponding to the combinations are pre-stored. Table 2below is an example of the table in which the values of the coefficientb corresponding to the value combinations of the vertical angle of viewVZ and the road width B are stored.

TABLE 2 Road width B Vertical angle B₁ ≦ B < B_(m−1) ≦ B < of view VZ B< B₁ B₂ . . . B_(m) B_(m) ≦ B VZ < VZ₁ b¹⁻¹ b¹⁻² . . . b_(1 −(m−1))b_(1−m) VZ₁ ≦ VZ < VZ₂ b²⁻¹ b²⁻² . . . b_(2−(m−1)) b_(2−m) . . . . . . .. . . . . . . . . . . VZ_(n−1) ≦ VZ < b_((n−1)−1) b_((n−1)−2) . . .b_((n−1)−(m−1)) b_((n−1)−m) VZ_(n)(180°) VZ_(n)(180°) ≦ VZ b_(n−1)b_(n−2) . . . b_(n−(m−1)) b_(n−m)

Note that the ways to determine the coefficients a, b are not limited tothose in the example described above. For example, in addition to theroad width, horizontal angle of view and vertical angle of view, theheight and width of an image sent out from the camera may be added tothe conditions for determining the coefficients a, b. Further, it is notnecessary to include the road width B in the conditions. Further, themagnification size may be determined by using the value representing thenon-blind spot area calculated at Op25. For example, the size of themagnified image may be set to fit into the non-blind spot area.

In this way, the magnifying unit 12 is notified of the magnificationsize (W2, H2) calculated at Op27. The magnifying unit 12 magnifies theroad mirror area in the original frames of the captured image such thatthe road mirror area is magnified to have the magnification size (W2,H2).

As described above with reference to FIGS. 10 to 12, the compositioncontrol unit 14 uses the imaging information on the camera 2 and thedistance L to calculate the imaging area, and uses the driverinformation and the distance L to calculate the visual field area. Then,the composition control unit 14 uses the value representing therelationship between the installation position of the camera 2 and theposition of the driver H1 to determine the positional relationshipbetween the imaging area and the visual field area. With the use of thispositional relationship, the composition control unit 14 may calculatethe value representing the non-blind spot area.

<Exemplary Display Image>

FIG. 13 is a diagram illustrating an example where a composite imageincluding the magnified image having the magnification size is displayedon the monitor 6. The example in FIG. 13 illustrates exemplary screenseach displayed on the monitor 6 when the vehicle 10 passes each positionin the course of entry into and passage of an intersection.

When the vehicle 10 has not arrived within a range apart from theintersection position K1 by the threshold value l₀ (when l₀<L), a screenD1 is displayed on the monitor 6. When the vehicle 10 enters within therange apart from the intersection position K1 by l₀, a screen D2 isdisplayed. In the screen D2, the road mirror portion is magnified and issuperimposed on the original captured image. Thereafter, until thevehicle 10 enters the intersecting road (period over which l₁<L≦l₀holds), the magnified image of the road mirrors becomes larger in sizeas the vehicle 10 moves closer to the intersection (screen D3). That is,the magnified image with the width W2 and height H2 calculated by theequations (12) and (13) is displayed. Then, during the period betweenthe entry of the vehicle 10 into the intersecting road and arrival atthe intersection position K1 (period over which 0<L≦l₁ holds), themagnified image is displayed in fixed size (W2, H2 of the equations(14), (15)) (screen D4). And when the vehicle 10 passes the intersectionposition K1 (0>L), the magnified image is not shown as in the screen D5.

As described above, because of the screen transitions illustrated inFIG. 13, the magnified image of road mirrors is superimposed on theoriginal captured image and is displayed in the size and at the positionthat allow the driver H1 to check the image easily in accordance withthe conditions of the vehicle 10. In particular, road conditions on theleft and right sides that are not directly observable to the driver H1are displayed in a single screen immediately after the entry of thevehicle into the intersection (immediately after L=l₁ holds). Thus, bysimply glancing at the monitor 6, the driver H1 may check the situationin the blind spots reflected on the road mirrors and the situation inthe blind spots captured by the camera 2.

Note that the screen transitions described above are one example and arenot limited to one described above. For example, when L<0 and l₀<L hold,an image captured by the camera 2 may not be displayed.

As described above, the driver H1 may check the situation in blind spotareas captured by both the camera 2 and road mirrors through a singleaction (action of checking the situation in the blind spot areas on themonitor 6). As a result, the burden on the driver H1 is reduced and thesituation that changes momentarily may be recognized with certainty.

Further, according to the image processor 3, by detecting the roadmirror area from the image captured by the camera 2, magnifying the areaand superimposing the magnified area on the non-blind spot area in theoriginal captured image, it is possible to display the composite imagein a single screen. Consequently, conditions of a bicycle, pedestrian orvehicle present in an area that cannot be captured by the camera 2 orroad mirrors alone may be presented to the driver H1. As a result, thenumber of collision accidents upon entry into intersections, accidentsmaking up the majority of traffic accidents, may be expected to decline.

<Modified Example of Calculation of Superimposition Position>

Here, a description will be given of a modified example of calculationof the imaging area (m1), the visual field area (m2) and the value (X)representing the positional relationship between the two areas, whichhave been described with reference to FIG. 11. In the exampleillustrated in FIG. 11, the optical axis of the camera 2 and the viewingdirection of the driver H1 (direction of eyes) are substantiallyparallel with each other and they are also parallel with the travelingdirection of the vehicle 10. In contrast, in this modified example, theoptical axis of the camera 2 and the viewing direction of the driver H1are not parallel with each other. FIG. 14 is a top view illustratingexemplary imaging area and visual field area in this modified examplewhen the vehicle 10 is about to enter an intersection. Variables L, m1,lm, n, α and β illustrated in FIG. 14 are the same as the variablesillustrated in FIG. 11. In the example illustrated in FIG. 14, thedirection of the driver H1's eyes is shifted from the travelingdirection by an angle γ on the horizontal plane. Here, a range betweenlines p8 and n8 is an area that is directly observable to the driver H1.The value of the angle γ may be pre-stored in the recording unit 16 ormay be set by the driver H1 through the navigation system 5, forexample. Further, information regarding the angle γ may be obtained froma vehicle-mounted device (not shown) for monitoring the driver H1's eyemovements through the vehicle information input unit 8.

Similarly to the example illustrated in FIG. 11, in the exampleillustrated in FIG. 14, ½ of the length of the area captured by thecamera 2 on the line q (=m1) may be calculated by the equation 2.

m1=L×tan(α/2)  (2)

In this modified example, it is assumed that the distance between anintersection point F1 of the line q and a line extending from the driverH1 in the traveling direction and the point of left end of the areadirectly observable to the driver H1 (intersection point of the lines p8and q) is m3. The composition control unit 14 calculates the value of m3as the value of the visual field area of the driver H1. Since the anglebetween a line connecting the point F1 and the driver H1 and the line p8is (γ+β/2), the following equation (16) holds true. Consequently, m3 maybe calculated by the following equation (17).

tan(γ+β/2)=m3/(L+l _(m))  (16)

m3=(L+l _(m))×tan(γ+β/2)  (17)

The composition control unit 14 uses m1 and m3 to calculate thepositional relationship between the imaging area and the visual fieldarea. Specifically, the composition control unit 14 calculates adistance X2 between the intersection point of the lines q and p6 and theintersection point of the lines q and p8. This is the distance betweenthe left end of the imaging area and the left end of the visual fieldarea. The value of X2 may be calculated by the following equations (18)to (20).

When l₀<L:

X2=0  (18)

When l₁<L≦l₀:

$\begin{matrix}\begin{matrix}{{X\; 2} = {{m\; 1} - \left( {{m\; 3} - n} \right)}} \\{= {\left\{ {L \times {\tan \left( {\alpha/2} \right)}} \right\} - \left\{ {{\left( {L + 1_{m}} \right) \times {\tan \left( {y + {\beta/2}} \right)}} - n} \right\}}}\end{matrix} & (19)\end{matrix}$

When 0<L≦l₁:

X2={l ₁×tan(α/2)}−{(l ₁ +l _(m))×tan(γ+β/2)−n}  (20)

With the use of m1, m3 and X2 calculated in this way, the compositioncontrol unit 14 calculates the value representing the non-blind spotarea in the image captured by the camera 2. For example, when it isassumed that the relationship between the width W1 of the captured imageand a length X_(PIX-2) between the left end of the captured image andthe left end of the non-blind spot area corresponds to the relationshipbetween the width (2×ml) of the imaging area and the distance X2, thefollowing equation (21) holds true. Consequently, X_(PIX-2) may becalculated by the following equation (22).

(2×m1):W1=X2:X _(PIX-2)  (21)

X _(PIX-2)=(X2×W1)/(2×m1)  (22)

The composition control unit 14 may calculate the value of X_(PIX-2) asthe value representing the non-blind spot area and notify thesuperimposition unit 13 of the value of X_(PIX-2) as the valuerepresenting the superimposition position. As a result, the output unit15 may produce a composite image in which the magnified image issuperimposed on the non-blind spot area in the captured image and outputthe composite image to the monitor 6.

In this modified example, the case where the eye direction of the driverH1 is shifted from the traveling direction towards the left side hasbeen described. By using the calculation method in this modifiedexample, it is possible to calculate the non-blind spot area even whenthe optical axis of the camera 2 is rotated by a certain angle(horizontal rudder angle) on the horizontal plane.

FIGS. 15A and 15B illustrate an example where the optical axis of thecamera 2 is rotated by a certain rudder angle. FIG. 15A is a top viewillustrating an example where the vehicle 10 is about to enter a road atan angle other than 90°. At the junction (T junction) illustrated inFIG. 15A, two roads merge at an angle λ. Here, as an example, it isassumed that the intersection point of center lines of these two mergingroads is a junction point K2. B denotes the width of the road on whichthe vehicle 10 is traveling and A2 denotes the width of the road towhich the vehicle is about to enter. L denotes the distance between thevehicle 10 and the junction position K2, l₁ denotes the distance betweenthe junction position K2 and the roadside, 12 denotes a threshold valuefor determining the staring point to display the magnified image and l₀denotes the distance between the junction position K2 and the startingpoint to display the magnified image. The distance l₁ is calculated byl₁=(A2/2)cos λ+(B/2)tan λ, for example. The road information such as thewidths A2 and B, the angle λ at which the two roads merge and thejunction position K2 may be obtained from the navigation system 5through the vehicle information input unit 8.

FIG. 15B is a magnified view illustrating the camera 2 portion of thevehicle 10. In FIG. 15B, the Y axis indicates the traveling direction ofthe vehicle 10 and a line J indicates the direction of optical axis ofthe camera 2 on the horizontal plane. On the horizontal plane, theoptical axis J of the camera 2 is rotated from the Y axis in ananti-clockwise direction by an angle ε. In other words, it could be saidthat the horizontal rudder angle of the camera 2 is ε.

Here, the rudder angle of the camera 2 may be controlled by other ECUmounted on the vehicle (e.g., a camera controlling ECU (not shown)). Asan example, this ECU may control the horizontal rudder angle ε of thecamera 2 in accordance with the angle λ between the road on which thevehicle 10 is traveling and the road into which the vehicle 10 is aboutto enter. For example, the horizontal rudder angle ε of the camera 2 iscontrolled such that the optical axis becomes perpendicular to the roadinto which the vehicle is about to enter.

For example, the image input unit 7 may obtain the informationindicating the horizontal rudder angle ε of the camera 2 from the ECU asone of the camera properties. By receiving the horizontal rudder angle εof the camera 2 from the image input unit 7, the composition controlunit 14 may use the angle to calculate the non-blind spot area.

The composition control unit 14 calculates the non-blind spot area bycalculating the imaging area when the camera 2 is rotated by the rudderangle ε and determining the positional relationship between thecalculated imaging area and the visual field area. Similarly to thecalculation method used in calculating the visual field area when thedirection of the driver H1's eyes is rotated by the angle γ describedabove, the composition control unit 14 may calculate the imaging areawhen the camera 2 is rotated by the horizontal rudder angle ε. Further,with the use of the equations (10) to (15), the composition control unit14 may calculate the size (W2, H2) of the magnified image appropriatelyin accordance with the distance L.

As described above, even when the optical axis of the camera 2 isrotated by the certain horizontal rudder angle on the horizontal plane,by calculating the non-blind spot area, it is possible to determine thesuperimposition position and the magnification size of the magnifiedimage.

<Calculation of Superimposition Position and Magnification Level whenVehicle 10 is heading for Curve>

Another modified example of the operation of the composition controlunit 14 will be described. Here, a description will be given of anexemplary calculation performed by the composition control unit 14 whenthe vehicle 10 is heading for a curve. FIG. 16 is a top viewillustrating an exemplary situation in which the vehicle 10 is headingfor a curve.

FIG. 16 illustrates a situation in which the vehicle 10 is about toapproach a curve with a radius of curvature R. As an example, it isassumed that a curve position K3 is an intersection point of the centerline of the road and a line bisecting the rotating angle (0) of thecurve. A denotes the width of the road on which the vehicle 10 istraveling, L denotes the distance between the vehicle 10 and the curveposition K3, l₁(l₁=(R+A/2)sin(θ/2)) denotes the distance between thecurve position K3 and the starting point of the curve, l₂ denotes athreshold value (fixed value) for determining the starting point todisplay the magnified image, and l₀ denotes the distance between thecurve position K3 and the starting point to display the magnified image.The road information such as the road width A, the rotating angle θ ofthe curve, the curve position K3 and the radius of curvature R of thecurve may be obtained from the navigation system 5 through the vehicleinformation input unit 8.

Similarly to the calculation of the non-blind spot area described withreference to FIGS. 11 and 12, the composition control unit 14 maycalculate the non-blind spot area in the captured image by determiningthe positional relationship between the imaging area and the visualfield area. Further, with the use of the following equations (23) to(28), the composition control unit 14 may calculate the size (W2, H2) ofthe magnified image appropriately in accordance with the distance L.

When l₀<L:

W2=0  (23)

H2=0  (24)

When l₁<L≦l₀:

W2=W1−(a×L)+(e×L)  (25)

H2=H1−(b×L)+(f×L)  (26)

When 0<L≦l₁:

W2=W1−(a×l ₁)+(e×l ₁)  (27)

H2=H1−(b×l ₁)+(f×l ₁)  (28)

Similarly to the coefficients a, b in the equations (12) to (15), thecoefficients a, b in the equations (25) to (28) may be determined basedon the camera properties and the road width A. The coefficients e, f aredetermined based on R of the curve. For example, for the coefficients e,f, values that become larger as R becomes smaller are predefined andstored in the recording unit 16. Specifically, the values of thecoefficients e, f may be determined with the use of a function thatdefines the values of the coefficients e, f that change in accordancewith R or a table in which the values of the coefficients e, frespectively corresponding to the values of R are stored.

In this way, even when the vehicle 10 approaches and passes a curve, thesuperimposition position and the magnification size of the magnifiedimage of road mirrors may be determined by calculating the non-blindspot area. Further, since the size of the magnified image is adjusted inaccordance with the curvature of curves, it is possible to display aroad mirror in large size at curves with a large curvature, that is, atsteep curves.

<Calculation for determining Magnification Level of Magnified Imageaccording to Frequency of Accidents>

In addition to the road information and the vehicle positioninformation, the composition control unit 14 may take a variety offactors into consideration when determining the magnification size ofthe magnified image. For example, the composition control unit 14 maydetermine the magnification size in accordance with the frequency ofoccurrence of accidents at the intersection into which the vehicle 10 isabout to enter. Here, as illustrated in FIG. 9, an exemplary calculationwhen the vehicle 10 is about to enter an intersection will be described.For example, the composition control unit 14 may obtain a valuerepresenting the frequency of occurrence of accidents at theintersection into which the vehicle 10 is about to enter from thenavigation system 5 through the vehicle information input unit 8.

For example, the value representing the frequency of occurrence ofaccidents is the number of accidents occurred in the vicinity of theintersection in the past 10 years. Alternatively, the value mayrepresent the frequency of season-by-season or period-by-periodoccurrence of accidents. Upon these values, the fact that the number ofaccidents is small during the summer but the incidence of accidentsincreases during the winter due to the road being covered with snow orthe number of accidents is small during the daytime but the numberincreases during the night time and early morning hours is reflected.

For example, with the use of the following equations (29) to (34), thecomposition control unit 14 may calculate the size (W2, H2) of themagnified image appropriately in accordance with the distance L.

When l₀<L:

W2=0  (29)

H2=0  (30)

When l₁<L ≦l₀:

W2=W1−(a×L)+(c×L)  (31)

H2=H1−(b×L)+(d×L)  (32)

When 0<L≦l₁:

W2=W1−(a×l ₁)+(c×l ₁)  (33)

H2=H1−(b×l ₁)+(d×l ₁)  (34)

Similarly to the coefficients a, b in the equations (12) to (15), thecoefficients a, b in the equations (29) to (34) may be determined basedon the camera properties and the road width A. Further, the coefficientsc, d are determined based on the frequency of occurrence of accidents atthe intersection into which the vehicle 10 is about to enter. Forexample, for the coefficients c, d, values that become larger as thefrequency of occurrence of accidents increases are predefined and storedin the recording unit 16. Specifically, the values of the coefficientsc, d may be determined with the use of a function that defines thevalues of the coefficients c, d that change in accordance with thefrequency of occurrence of accidents or a table in which the values ofthe coefficients c, d respectively corresponding to the values of thefrequency of occurrence of accidents are stored.

In this way, the magnification size of the magnified image of a roadmirror may be determined in accordance with the frequency of occurrenceof accidents at the intersection into which the vehicle 10 is about toenter. As a result, at an intersection with a high frequency ofoccurrence of accidents, the magnified image of a road mirror isincreased in size, so that an image easily recognizable by the driver H1may be displayed on the monitor 6.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiment(s) of the presentinvention has (have) been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

1. An image processor comprising: an image input unit that obtains animage captured by an imaging device installed on a vehicle; a vehicleinformation input unit that obtains a distance between the vehicle and aroad junction or curve in a traveling direction of the vehicle based oninformation received from a vehicle-mounted device installed in thevehicle; a recording unit in which imaging information indicatingproperties of the imaging device and driver information regarding avisual field of a driver of the vehicle are stored; a magnifying objectrecognition unit that recognizes a certain magnifying object in thecaptured image; and a composing unit that produces, when the magnifyingobject recognition unit recognizes the certain magnifying object, acomposite image of a magnified image of the magnifying object and thecaptured image, wherein the composing unit uses the distance obtained bythe vehicle information input unit as well as the imaging informationand the driver information stored in the recording unit to determine anon-blind spot area in the captured image that does not include an imageof a blind spot for the driver, and produces the composite image suchthat the magnified image is superimposed on the non-blind spot area. 2.The image processor according to claim 1, wherein the composing unituses the distance between the vehicle and the junction or the curve andthe imaging information to calculate an imaging area to be captured bythe imaging device in a vicinity of a position of the junction or thecurve, uses the distance between the vehicle and the junction or thecurve and the driver information to calculate a visual field area to bein the visual field of the driver in the vicinity of the position of thejunction or the curve, and uses a positional relationship between theimaging area and the visual field to determine a position of thenon-blind spot area in the captured image.
 3. The image processoraccording to claim 1, wherein the composing unit uses the distanceobtained by the vehicle information input unit to determine amagnification level of the magnified image.
 4. The image processoraccording to claim 2, wherein the image input unit further obtains ahorizontal rudder angle of the imaging device at a time of capturing thecaptured image, and the composing unit uses the horizontal rudder angleto calculate the imaging area.
 5. The image processor according to claim1, wherein the vehicle information input unit further obtains a valuerepresenting a curvature of the curve in the traveling direction of thevehicle, and the composing unit uses the curvature to determine amagnification level of the magnified image.
 6. The image processoraccording to claim 1, wherein the vehicle information input unit furtherobtains information indicating a frequency of occurrence of accidents atthe junction or the curve, and the composing unit uses the frequency ofoccurrence of accidents to determine a magnification level of themagnified image.
 7. The image processor according to claim 1, whereinthe image processor works in conjunction with a navigation systeminstalled in the vehicle, and the vehicle information input unit obtainsthe distance between the vehicle and the road junction or the curve inthe traveling direction of the vehicle from the navigation systeminstalled in the vehicle.
 8. The image processor according to claim 7,wherein the vehicle information input unit further obtains informationindicating presence or absence of the magnifying object at the junctionor the curve from the navigation system, and the magnifying objectrecognition unit recognizes the certain magnifying object in thecaptured image when the certain magnifying object is present at thejunction or the curve.
 9. The image processor according to claim 1,wherein the magnifying object recognition unit recognizes a certainmagnifying object in the captured image when the distance becomes equalto or less than a certain distance.
 10. A non-transitory recordingmedium storing an image processing program causing a computer to performprocessing of: obtaining an image captured by an imaging deviceinstalled on a vehicle; obtaining a distance between the vehicle and aroad junction or curve in a traveling direction of the vehicle based oninformation received from a vehicle-mounted device installed in thevehicle; obtaining imaging information indicating properties of theimaging device and driver information regarding a visual field of adriver of the vehicle stored in a recording unit accessible to thecomputer; recognizing a certain magnifying object in the captured image;and producing, when the certain magnifying object is recognized in themagnifying object recognition, a composite image of a magnified image ofthe magnifying object and the captured image, wherein in the productionof the composite image, the obtained distance as well as the obtainedimaging information and driver information are used to determine anon-blind spot area in the captured image that does not include an imageof a blind spot for the driver, and the composite image is produced suchthat the magnified image is superimposed on the non-blind spot area. 11.A vehicle-mounted terminal capable of working in conjunction with animaging device installed on a vehicle, the vehicle-mounted terminalcomprising: a navigation system having a function of identifying aposition of the vehicle and including a map data recording unit in whichroad information including a position of each road junction or curve isstored; an image input unit that obtains an image captured by theimaging device; a vehicle information input unit that obtains a distancebetween the vehicle and a road junction or curve in a travelingdirection of the vehicle with the use of the position of the vehicleidentified by the navigation system and the road information; arecording unit in which imaging information indicating properties of theimaging device and driver information regarding a visual field of adriver of the vehicle are stored; a magnifying object recognition unitthat recognizes a certain magnifying object in the captured image; acomposing unit that produces, when the magnifying object recognitionunit recognizes the certain magnifying object, a composite image of amagnified image of the magnifying object and the captured image; and adisplay unit that displays the composite image produced by the composingunit.